Speech and computer vision-based control

ABSTRACT

The present disclosure relates to a method for controlling a digital photography system. The method includes obtaining, by a device, image data and audio data. The method also includes identifying one or more objects in the image data and obtaining a transcription of the audio data. The method also includes controlling a future operation of the device based at least on the one or more objects identified in the image data, and the transcription of the audio data.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No.62/202,488, filed Aug. 7, 2015, the contents of which are incorporatedby reference.

TECHNICAL FIELD

This specification generally relates to speech recognition and computervision.

BACKGROUND

Digital cameras, which are devices for recording images such as stillphotographs or videos, are used in photography. Because photographersoften take pictures of other people, photography is seldom a purely solophenomenon. Rather, such as in the instances where a photographer takespictures at a party, or where a shared camera is left on a table at aparty for many people to use, photography is often used to enhance alarger social experience.

SUMMARY

According to one implementation, this specification describes a processfor controlling an operation of a digital camera device based at leaston speech commands that are uttered by a photographer and/or by humansubjects of a photo, as well as one or more features, e.g., gestures,faces, or objects, in a digital camera image. For example, a user canspeak a command to instruct a camera to automatically take or share,e.g., by uploading to a social network or picture storage site, picturesin future circumstances when a certain visual feature is present withinthe field of view of the camera, and the camera will then take a picturewhenever that feature is identified. As another example, a voice commandcan instruct a camera to not take or share a picture when a certainvisual feature is present within the field of view of the camera. Inthis manner, a set of rules for automatically taking pictures can begenerated by one or more users, and the digital camera may automaticallytake pictures based on the set of rules without further contemporaneousor direct commands from the one or more users.

The present disclosure relates to a method of obtaining, by a device,(i) image data and (ii) audio data; identifying one or more objects inthe image data; obtaining a transcription of the audio data; andcontrolling a future operation of the device based at least on (i) theone or more objects identified in the image data, and (ii) thetranscription of the audio data.

The present disclosure also relates to a system comprising one or morecomputers and one or more storage devices storing instructions that areoperable, when executed by the one or more computers, to cause the oneor more computers to perform operations comprising: obtaining, by adevice, (i) image data and (ii) audio data; identifying one or moreobjects in the image data; obtaining a transcription of the audio data;and controlling a future operation of the device based at least on (i)the one or more objects identified in the image data, and (ii) thetranscription of the audio data.

In addition, the present disclosure relates to a computer-readablemedium storing software comprising instructions executable by one ormore computers which, upon such execution, cause the one or morecomputers to perform operations comprising: obtaining, by a device, (i)image data and (ii) audio data; identifying one or more objects in theimage data; obtaining a transcription of the audio data; and controllinga future operation of the device based at least on (i) the one or moreobjects identified in the image data, and (ii) the transcription of theaudio data.

Implementations may include one or more of the following features.Controlling a future operation of the device may comprise determiningwhether to capture future image data, or determining whether toautomatically upload future generated image data to cloud storage.Identifying one or more objects in the image data may comprise at leastone of identifying a person using face detection, identifying a gestureperformed by a person in the image, or detecting an action performed bya person in the image. The image data and the audio data may begenerated by the device. A set of one or more rules may be generated,where the controlling of a future operation of the device is based onthe set of one or more rules. The transcription of the audio data may beobtained using automated speech recognition. The one or more objects inthe image data may be identified using computer vision.

Advantageous implementations may include one or more of the followingfeatures. The combination of computer vision and speech recognition mayenable collaborative photography techniques. Speech recognition mayprovide a separate control and information stream that cansynergistically augment computer vision. The system and techniquesdescribed may allow individuals other than the device user to controlthe device as they desire. The system and techniques also may allowindividuals to tailor captured images to their personal preferences. Thenumber of unwanted captured images may be reduced, reducing storagerequirements and increasing network bandwidth, as the system iscontrolled to only take or share captured images that satisfy spokencriteria. As the device automatically captures desired images based on aset of rules, the user may not need to spend as much time manuallycontrolling the device to capture images.

The details of one or more implementations of the subject matterdescribed in this specification are set forth in the accompanyingdrawings and the description below. Other potential features, aspects,and advantages of the subject matter will become apparent from thedescription, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1F are diagrams of example digital photography systems.

FIG. 2 is a flow chart illustrating an example process for controlling adigital photography system.

FIG. 3 is a diagram of an example digital photography system.

FIG. 4 shows an example of a computing device and an example of a mobilecomputing device that can be used to implement the techniques describedhere.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

Traditional cameras have been operated solely based on the manual inputsof photographers that are holding or otherwise controlling the camera.As such, photographers must typically be trained to operate a camera,and must also remain cognizant of the picture taking environment, so asto take pictures of the desired subjects or situations. Certain videocameras may record only when movement is detected by a motion sensor.However, as photography has become less specialized and more social,people who are typically the subjects of a picture have shown additionalinterest in taking or sharing pictures of themselves or others, or ofcertain situations, without instructing a photographer and withoutreceiving specialized photography training.

Accordingly, the present disclosure describes techniques for usingspeech recognition and computer vision to instruct a future operation ofa camera. The camera may receive speech commands from a user or otherperson, and the camera may automatically capture an image if a featurein the image corresponds to the speech command. Speech recognitionallows people other than camera's owner to control picture taking byissuing voice commands. Computer vision allows the camera to identifywho is in the frame and who is speaking. In some cases, speechrecognition may provide an entirely separate control and informationstream that synergistically augments computer vision. By combiningspeech recognition and computer vision, the disclosed techniques canallow picture taking to be much more of a shared, collaborativeexperience.

FIGS. 1A-1D show diagrams of examples of a digital photography system100. The digital photography system 100 includes an example digitalcamera 102 that is generating an image of one or more subjects 104. Thecamera 102 includes a display 108 that may show an image of the subjects104 as well as additional information. One or more of the subjects 104issues an audio command 106 that is received by the camera 102. Thecamera 102 interprets the audio command 106 and visual informationassociated with the subjects 104 to determine whether and when tocapture an image of the subjects 104. The interpretation 110 showsexample audio cues and visual cues that may be used to determine one ormore rules for capturing images.

The digital camera 102 may be any suitable device through which an imagecan be obtained. For example, the digital camera 102 may include astandalone camera or a camera implemented in a smartphone, tabletcomputer, portable computer, desktop computer, or other device. Thedisplay 108 may be an LCD screen, touchscreen, or other type of displaythat may show images and/or information. In some implementations, thedigital camera 102 does not include a display 108, or is coupled to adisplay 108 that is separate from the digital camera 102. In someimplementations, the digital camera 102 is worn by a single user, forexample around the neck or clipped to the shirt pocket. The digitalcamera 102 may include one or more microphones. In some implementations,the digital camera 102 may be continuously obtaining images and/orrunning computer vision.

FIG. 1A shows an example of digital photography system 100. In FIG. 1A,the subject 104 include two persons, a man 105 a and a woman 105 b. Thewoman 105 bhas uttered the speech command 106, “Take pictures of me andmy boyfriend.” As shown on the camera display 108, the camera 102 hasidentified the speech utterance 106 as well as the man 105 a as “John”and the woman 10 b as “Marsha.”

Example interpretation 110 shows how the camera 102 may interpret thespeech command 106 and visual information, such as that shown on display108. The camera 102 may be able to identify “John” as the boyfriend of“Marsha” using face detection, metadata, contact information, socialmedia, or other techniques. In some cases, the camera 102 may categorizethese subjects as visual cues. The camera 102 may also interpret thespeech command 106, “Take pictures of me and my boyfriend.” For example,the camera 102 may identify “me” as the woman 150 b and “my boyfriend”as the man 105 a. The camera 102 may use speech recognition techniquesto transcribe the speech command 106.

As example interpretation 110 shows, the camera 102 may interpret thespeech command 106 and visual information into a new rule “Take picturewhen image shows John and Marsha.” In this manner, the camera 102 may bedirected to automatically capture images whenever both John and Marshaare identified in the same image, e.g., when both John and Marsha arewithin the field of view of the camera.

FIG. 1B shows a second example of digital photography system 100. InFIG. 1B, the subject 104 is a single person, and the speech command 106is “Take pictures when I give thumbs-up.” In this example, the camera102 may identify the subject 104 as “John,” and also identify that thesubject 104 is giving a “thumbs-up” sign. The example interpretation 110shows that the camera 102 has identified the speaker as John as and thatJohn is giving a thumbs-up sign.

Based on these visual and audio inputs, the camera 102 has generated therule “Take picture when image shows John and John thumbs-up.” In thismanner, the camera 102 may be directed to automatically capture imagesthat show John whenever John is giving a thumbs-up. While the exampleshown in FIG. 1B uses a thumbs-up as a visual cue, in otherimplementations, other visual signifiers may be identified, such asother types of body language, gestures, facial features such as smiling,blinking, etc., laughter, crying, or other sounds, activities such aswalking, jumping, etc., or other visual signifiers.

FIG. 1C shows a third example of digital photography system 100. In FIG.1C, the subject 104 is a single person, and the speech command 106 is“Take pictures of people in striped shirts.” In this example, the camera102 may identify that the subject 104 is wearing a striped shirt. Theexample interpretation 110 shows that the camera 102 has identified thesubject 104 as wearing a striped shirt.

Based on these inputs, the camera 102 has generated the rule “Takepicture when image shows person in striped shirt.” In this manner, thecamera 102 may be directed to automatically capture images that show aperson wearing a striped shirt. In some cases, the camera 102 may take apicture of a person wearing a striped shirt that is not the same personas subject 104. In this manner, the camera may take pictures of personswith specific features, such as wearing a certain color of hat, acertain logo on clothing, certain color hair, or other features.

FIG. 1D shows a fourth example of digital photography system 100. InFIG. 1D, the subject 104 is a landmark 105 c and single person 105 d,and the speech command 106 is “Take pictures of me and the EiffelTower.” In this example, the camera 102 may identify that the speaker isthe person 105 d and that the landmark 105 c is the Eiffel Tower. Theexample interpretation 110 shows that the camera 102 has identified theperson 105 d as “John” and the landmark 105 c is the Eiffel Tower.

Based on these inputs, the camera 102 has generated the rule “Takepicture when the image shows John and the Eiffel Tower.” In this manner,the camera 102 may be directed to automatically capture images that showJohn and the Eiffel Tower. In some cases, the camera 102 may be directedto take a picture of a person who is not the speaker. In this manner,the camera 102 may identify and take pictures of non-person subjectssuch as buildings, landmarks, pets or other animals, objects, naturalfeatures, or other non-person subjects.

FIG. 1E shows a fifth example of digital photography system 100. In FIG.1E, the subject 104 is a single person, and the speech command 106 is“Don't take pictures of me.” In this example, the camera 102 mayidentify that the subject 104 is “John.” The camera 102 has generatedthe rule “Do not take picture when image shows John.” In this manner,the camera 102 may be directed to not capture images that show John. Inthis manner, the camera 102 may be directed to not capture images thatshow certain people, objects, locations, or other features. For example,the speech command may be “Do not take a picture of my daughter,” “Donot take a picture of my car,” or another command.

FIG. 1F shows a sixth example of digital photography system 100. In FIG.1F, the subject 104 is single person 105 e holding food 105 f, and thespeech command 106 is “Don't take pictures of me while I'm eating.” Inthis example, the camera 102 may identify that the speaker is the person105 e and that the person is holding food 105 f. The exampleinterpretation 110 shows that the camera 102 has identified the person105 e as “John” and identified the food 105 f.

In light of these inputs, the camera 102 has generated the rule “Do nottake picture when image shows John and good.” In this manner, the camera102 may be directed to not capture images that show both John and food.In some cases, the camera 102 may be directed to take a picture of aperson who is not the speaker. In this manner, picture-taking rules maybe generated based on persons or objects identified in an image.

Alternatively, the speech command 106 may be an instantaneous command totake a picture. The camera 102 may interpret such a speech command 106and determine to capture an image in response. For example, a person inview of the camera may say “cheese!” or “snap a picture!” or “shootthis!” as the speech command 106. The camera 102 may identify that aperson is in view of the camera 102 and may be directed to capture animage in response to the speech command 106.

FIG. 2 is a flow chart illustrating an example process 200 forcontrolling a digital photography system. The example process 200 may beimplemented, for example, by some or all of system 100. At 202, a deviceobtains image data and audio data. For example, the audio data mayinclude a data file of a waveform, an audio file, or other data typethat encodes an utterance. The image data may include a data file of animage, a data file of a video, or other data that encodes an image orvideo. The device may be a digital camera, a portable computer,smartphone, or other device as described previously. In someimplementations, the device generates the audio data and the video data.

At 204, one or more objects in the image data are identified. Theobjects can be people, structures, animals, gestures, or other featuresas described previously. In some cases, an object is identified usingcomputer vision. At 206, a transcription of the audio data is obtained.The transcription may be obtained, for example, using a speechrecognizer. A speech recognizer may use one or more speech recognizercomponents in generating transcriptions of audio data such as anacoustic model, a language model, or another type of speech recognizercomponent

At 208, a future operation of the device is controlled based on at leastan object identified in the image data and the transcription of theaudio data. Controlling a future operation of the device may includedetermining whether to capture future image data. For example, imagedata obtained in the future may be captured if the image data includesan image of a particular object.

In other examples, controlling a future operation of the device mayinclude determining whether to automatically upload future generatedimage data to cloud storage. For example, whether to store the datalocally or whether to store the data in cloud storage can be determinedbased on instructions within the audio data and one or more objectsidentified in the image data. In some cases, controlling a futureoperation of the device may include controlling parameters or featuresof the device. For instance, the flash, the focus, the aperture, orother parameters may be controlled. As an illustrative example, thedevice could obtain a transcription of “Use the flash when taking apicture of me,” indicating that the camera should turn the flash on whenthe speaker is identified in an image. Controlling the future operationof the device can also be based on a set of one or more rules.

FIG. 3 is a diagram of an example digital photography system 300. Thesystem 300 may be implemented by, for example, the system 100 shown inFIG. 1 or the process shown in FIG. 2. The system 300 may also beincluded as part of a device such as a camera, portable computer,smartphone, or other device as described previously. The system 300includes a microphone 302. The microphone 302 receives audio energy,e.g. speech or other noise, and transmits the audio energy as a signalto a speech recognizer 304. The microphone 302 may transmit the audioenergy as a signal, data, a data file, or in another form.

The speech recognizer 304 receives the audio signal from the microphone302 and generates a transcript of speech present in the audio signal. Insome cases, the speech recognizer 304 communicates with a speechrecognition server to generate a transcription. The speech recognitionserver is not shown in FIG. 3. The speech recognition server can receivespeech recognition requests from a device, and, using one or more speechmodels, provide speech transcriptions back to the device.

The system 300 also includes an image sensor 306. The image sensor 306may be a digital camera sensor, CCD camera sensor, or other type ofimage sensor. The image sensor 306 obtains image data and transmits theimage data to a video feature detector 308. The video feature detector308 may identify features within the image data received from the imagesensor 306. For example, the video feature detector may detect and/oridentify faces within the image data.

Both the speech recognizer 304 and the video feature detector 308transmit data to the rules interpretation engine 310. For instance, thespeech recognizer 304 may transmit a transcript and/or transcriptmetadata to the rules interpretation engine 310, and the video featuredetector 308 may transmit information associated with detected featuresto the rules interpretation engine 310. The rules interpretation engine310 receives the data from the speech recognizer 304 and video featuredetector 308 and uses the data to generate a set of one or more rulescontrolling future device operation.

The set of rules may be transmitted to a smart device rules managementengine 312 that stores and manages sets of rules. For example, the rulesmanagement engine 312 may organize sets of rules, classify sets ofrules, check sets of rules for inconsistencies, or other managementoperations. The rules management engine 312 interfaces with the smartdevice controller 314 to implement the rules controlling the device. Forexample, the smart device controller 314 may interact with the rulesmanagement engine 312 to determine that one or more of the rules issatisfied, and to control the device accordingly.

FIG. 4 shows an example of a computing device 400 and an example of amobile computing device that can be used to implement the techniquesdescribed here. The computing device 400 is intended to representvarious forms of digital computers, such as laptops, desktops,workstations, personal digital assistants, servers, blade servers,mainframes, and other appropriate computers. The mobile computing deviceis intended to represent various forms of mobile devices, such aspersonal digital assistants, cellular telephones, smart-phones, digitalcameras, and other similar computing devices. The components shown here,their connections and relationships, and their functions, are meant tobe exemplary only, and are not meant to limit implementations of theinventions described and/or claimed in this document.

The computing device 400 includes a processor 402, a memory 404, astorage device 406, a high-speed interface 408 connecting to the memory404 and multiple high-speed expansion ports 410, and a low-speedinterface 412 connecting to a low-speed expansion port 414 and thestorage device 406. Each of the processor 402, the memory 404, thestorage device 406, the high-speed interface 408, the high-speedexpansion ports 410, and the low-speed interface 412, are interconnectedusing various busses, and may be mounted on a common motherboard or inother manners as appropriate. The processor 402 can process instructionsfor execution within the computing device 400, including instructionsstored in the memory 404 or on the storage device 406 to displaygraphical information for a GUI on an external input/output device, suchas a display 416 coupled to the high-speed interface 408. In otherimplementations, multiple processors and/or multiple buses may be used,as appropriate, along with multiple memories and types of memory. Also,multiple computing devices may be connected, with each device providingportions of the necessary operations, e.g., as a server bank, a group ofblade servers, or a multi-processor system.

The memory 404 stores information within the computing device 400. Insome implementations, the memory 404 is a volatile memory unit or units.In some implementations, the memory 404 is a non-volatile memory unit orunits. The memory 404 may also be another form of computer-readablemedium, such as a magnetic or optical disk.

The storage device 406 is capable of providing mass storage for thecomputing device 400. In some implementations, the storage device 406may be or contain a computer-readable medium, such as a floppy diskdevice, a hard disk device, an optical disk device, or a tape device, aflash memory or other similar solid state memory device, or an array ofdevices, including devices in a storage area network or otherconfigurations. A computer program product can be tangibly embodied inan information carrier. The computer program product may also containinstructions that, when executed, perform one or more methods, such asthose described above. The computer program product can also be tangiblyembodied in a computer- or machine-readable medium, such as the memory404, the storage device 406, or memory on the processor 402.

The high-speed interface 408 manages bandwidth-intensive operations forthe computing device 400, while the low-speed interface 412 manageslower bandwidth-intensive operations. Such allocation of functions isexemplary only. In some implementations, the high-speed interface 408 iscoupled to the memory 404, the display 416, e.g., through a graphicsprocessor or accelerator, and to the high-speed expansion ports 410,which may accept various expansion cards, not shown. In theimplementation, the low-speed interface 412 is coupled to the storagedevice 406 and the low-speed expansion port 414. The low-speed expansionport 414, which may include various communication ports, e.g., USB,Bluetooth, Ethernet, wireless Ethernet or others, may be coupled to oneor more input/output devices, such as a keyboard, a pointing device, ascanner, or a networking device such as a switch or router, e.g.,through a network adapter.

The computing device 400 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 420, or multiple times in a group of such servers. Inaddition, it may be implemented in a personal computer such as a laptopcomputer 422. It may also be implemented as part of a rack server system424. Alternatively, components from the computing device 400 may becombined with other components in a mobile device, not shown, such as amobile computing device 450. Each of such devices may contain one ormore of the computing device 400 and the mobile computing device 450,and an entire system may be made up of multiple computing devicescommunicating with each other.

The mobile computing device 450 includes a processor 452, a memory 464,an input/output device such as a display 454, a communication interface466, and a transceiver 468, among other components. The mobile computingdevice 450 may also be provided with a storage device, such as amicro-drive or other device, to provide additional storage. Each of theprocessor 452, the memory 464, the display 454, the communicationinterface 466, and the transceiver 468, are interconnected using variousbuses, and several of the components may be mounted on a commonmotherboard or in other manners as appropriate.

The processor 452 can execute instructions within the mobile computingdevice 450, including instructions stored in the memory 464. Theprocessor 452 may be implemented as a chipset of chips that includeseparate and multiple analog and digital processors. The processor 452may provide, for example, for coordination of the other components ofthe mobile computing device 450, such as control of user interfaces,applications run by the mobile computing device 450, and wirelesscommunication by the mobile computing device 450.

The processor 452 may communicate with a user through a controlinterface 458 and a display interface 456 coupled to the display 454.The display 454 may be, for example, a TFT (Thin-Film-Transistor LiquidCrystal Display) display or an OLED (Organic Light Emitting Diode)display, or other appropriate display technology. The display interface456 may comprise appropriate circuitry for driving the display 454 topresent graphical and other information to a user. The control interface458 may receive commands from a user and convert them for submission tothe processor 452. In addition, an external interface 462 may providecommunication with the processor 452, so as to enable near areacommunication of the mobile computing device 450 with other devices. Theexternal interface 462 may provide, for example, for wired communicationin some implementations, or for wireless communication in otherimplementations, and multiple interfaces may also be used.

The memory 464 stores information within the mobile computing device450. The memory 464 can be implemented as one or more of acomputer-readable medium or media, a volatile memory unit or units, or anon-volatile memory unit or units. An expansion memory 474 may also beprovided and connected to the mobile computing device 450 through anexpansion interface 472, which may include, for example, a SIMM (SingleIn Line Memory Module) card interface. The expansion memory 474 mayprovide extra storage space for the mobile computing device 450, or mayalso store applications or other information for the mobile computingdevice 450. Specifically, the expansion memory 474 may includeinstructions to carry out or supplement the processes described above,and may include secure information also. Thus, for example, theexpansion memory 474 may be provide as a security module for the mobilecomputing device 450, and may be programmed with instructions thatpermit secure use of the mobile computing device 450. In addition,secure applications may be provided via the SIMM cards, along withadditional information, such as placing identifying information on theSIMM card in a non-hackable manner.

The memory may include, for example, flash memory and/or NVRAM memory(non-volatile random access memory), as discussed below. In someimplementations, a computer program product is tangibly embodied in aninformation carrier. The computer program product contains instructionsthat, when executed, perform one or more methods, such as thosedescribed above. The computer program product can be a computer- ormachine-readable medium, such as the memory 464, the expansion memory474, or memory on the processor 452. In some implementations, thecomputer program product can be received in a propagated signal, forexample, over the transceiver 468 or the external interface 462.

The mobile computing device 450 may communicate wirelessly through thecommunication interface 466, which may include digital signal processingcircuitry where necessary. The communication interface 466 may providefor communications under various modes or protocols, such as GSM voicecalls (Global System for Mobile communications), SMS (Short MessageService), EMS (Enhanced Messaging Service), or MMS messaging (MultimediaMessaging Service), CDMA (code division multiple access), TDMA (timedivision multiple access), PDC (Personal Digital Cellular), WCDMA(Wideband Code Division Multiple Access), CDMA2000, or GPRS (GeneralPacket Radio Service), among others. Such communication may occur, forexample, through the transceiver 468 using a radio-frequency. Inaddition, short-range communication may occur, such as using aBluetooth, WiFi, or other such transceiver (not shown). In addition, aGPS (Global Positioning System) receiver module 470 may provideadditional navigation- and location-related wireless data to the mobilecomputing device 450, which may be used as appropriate by applicationsrunning on the mobile computing device 450.

The mobile computing device 450 may also communicate audibly using anaudio codec 460, which may receive audio information and convert it tousable digital information. The audio codec 460 may likewise generateaudible sound for a user, such as through a speaker, e.g., in a handsetof the mobile computing device 450. Such sound may include sound fromvoice telephone calls, may include recorded sound, e.g., voice messages,music files or others, and may also include sound generated byapplications operating on the mobile computing device 450.

The mobile computing device 450 may also include an imaging system (notshown), which may receive images and convert the images to usabledigital information. The image system may include an image sensor suchas a CCD camera or other digital image sensor. In some implementations,the imaging system receives images continuously. The imaging system maycapture image data, store image data, upload image data to cloudstorage, or otherwise maintain image data. The imaging system mayinclude one or more video codecs to generate image data. The image datamay include still images and/or video. The imagine system may interfacewith display 454 on the mobile computing device 450 to show image dataon the display 454.

The mobile computing device 450 may be implemented in a number ofdifferent forms, as shown in the figure. For example, it may beimplemented as a cellular telephone 480. It may also be implemented aspart of a smart-phone 482, digital camera, personal digital assistant,or other similar mobile device.

Various implementations of the systems and techniques described here maybe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware, firmware, software, and/or combinations thereof.These various implementations may include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

These computer programs, also known as programs, software, softwareapplications or code, include machine instructions for a programmableprocessor, and may be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium” and“computer-readable medium” refer to any computer program product,apparatus and/or device, e.g., magnetic discs, optical disks, memory,Programmable Logic Devices (PLDs), used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as a machine-readable signal.The term “machine-readable signal” refers to any signal used to providemachine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniquesdescribed here may be implemented on a computer having a display device,e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitorfor displaying information to the user, and a keyboard and a pointingdevice, e.g., a mouse or a trackball, by which the user may provideinput to the computer. Other kinds of devices may be used to provide forinteraction with a user as well; for example, feedback provided to theuser may be any form of sensory feedback, e.g., visual feedback,auditory feedback, or tactile feedback, and input from the user may bereceived in any form, including acoustic, speech, or tactile input.

The systems and techniques described here may be implemented in acomputing system that includes a back end component, e.g., as a dataserver, or that includes a middleware component, e.g., an applicationserver, or that includes a front end component, e.g., a client computerhaving a graphical user interface or a Web browser through which a usermay interact with an implementation of the systems and techniquesdescribed here, or any combination of such back end, middleware, orfront end components. The components of the system may be interconnectedby any form or medium of digital data communication, e.g., acommunication network. Examples of communication networks include alocal area network (“LAN”), a wide area network (“WAN”), and theInternet.

The computing system may include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

A number of embodiments have been described. Nevertheless, it will beunderstood that various modifications may be made without departing fromthe spirit and scope of the invention. In addition, the logic flowsdepicted in the figures do not require the particular order shown, orsequential order, to achieve desirable results. In addition, other stepsmay be provided, or steps may be eliminated, from the described flows,and other components may be added to, or removed from, the describedsystems. Accordingly, other embodiments are within the scope of thefollowing claims.

What is claimed is:
 1. A computer-implemented method comprising:obtaining, by a device, (i) image data and (ii) audio data; identifyingone or more objects in the image data; obtaining a transcription of theaudio data; and controlling a future operation of the device based atleast on (i) the one or more objects identified in the image data, and(ii) the transcription of the audio data.
 2. The method of claim 1,wherein controlling a future operation of the device comprisesdetermining whether to capture future image data.
 3. The method of claim1, wherein controlling a future operation of the device comprisesdetermining whether to automatically upload future generated image datato cloud storage.
 4. The method of claim 1, wherein identifying one ormore objects in the image data comprises at least one of identifying aperson using face detection, identifying a gesture performed by a personin the image, or detecting an action performed by a person in the image.5. The method of claim 1, further comprising generating, by the device,the image data and the audio data.
 6. The method of claim 1, furthercomprising generating a set of one or more rules, wherein thecontrolling of a future operation of the device is based on the set ofone or more rules.
 7. The method of claim 1, wherein the device is acamera.
 8. The method of claim 1, wherein the transcription of the audiodata is obtained using automated speech recognition.
 9. The method ofclaim 1, wherein the one or more objects in the image data is identifiedusing computer vision.
 10. A system comprising: one or more computersand one or more storage devices storing instructions that are operable,when executed by the one or more computers, to cause the one or morecomputers to perform operations comprising: obtaining, by a device, (i)image data and (ii) audio data; identifying one or more objects in theimage data; obtaining a transcription of the audio data; and controllinga future operation of the device based at least on (i) the one or moreobjects identified in the image data, and (ii) the transcription of theaudio data.
 11. The system of claim 10, wherein controlling a futureoperation of the device comprises determining whether to capture futureimage data.
 12. The system of claim 10, wherein controlling a futureoperation of the device comprises determining whether to automaticallyupload future generated image data to cloud storage.
 13. The system ofclaim 10, wherein identifying one or more objects in the image datacomprises at least one of identifying a person using face detection,identifying a gesture performed by a person in the image, or detectingan action performed by a person in the image.
 14. The system of claim10, further comprising generating, by the device, the image data and theaudio data.
 15. The system of claim 10, further comprising generating aset of one or more rules, wherein the controlling of a future operationof the device is based on the set of one or more rules.
 16. The systemof claim 10, wherein the device is a camera.
 17. The system of claim 10,wherein the transcription of the audio data is obtained using automatedspeech recognition.
 18. The system of claim 10, wherein the one or moreobjects in the image data is identified using computer vision.
 19. Acomputer-readable medium storing software comprising instructionsexecutable by one or more computers which, upon such execution, causethe one or more computers to perform operations comprising: obtaining,by a device, (i) image data and (ii) audio data; identifying one or moreobjects in the image data; obtaining a transcription of the audio data;and controlling a future operation of the device based at least on (i)the one or more objects identified in the image data, and (ii) thetranscription of the audio data.
 20. The system of claim 19, furthercomprising generating a set of one or more rules, wherein thecontrolling of a future operation of the device is based on the set ofone or more rules.